Harmonized Dense Knowledge Distillation Training for Multi-Exit Architectures

Multi-exit architectures, in which a sequence of intermediate classifiers are introduced at different depths the feature layers, perform adaptive computation by early exiting ``easy" samples to speed up inference. In this paper, novel Harmonized Dense Knowledge Distillation (HDKD) training method for multi-exit architecture is designed encourage each exit flexibly learn from all its later exits. particular, general dense knowledge distillation objective proposed incorporate possible beneficial supervision information learning, where harmonized weighting scheme multi-objective optimization problem consisting classification loss and loss. A bilevel algorithm alternatively updating weights multiple objectives network parameters. Specifically, parameters optimized with respect performance on validation set gradient descent. Experiments CIFAR100 ImageNet show that HDKD strategy harmoniously improves state-of-the-art neural networks. Moreover, does not require within modifications can be effectively combined other previously-proposed techniques further boosts performance.